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RELIABILITY  OF  ASSOCUTE  RATINGS  OF  PERFORHANCE  POTENTIAI,  BY 
AVIATORS 


AR^fY 


BACKGROUND 

In  response  to  a  TRADOC  request,  the  Fort  Rucker  Field  Unit  of  the 
Array  Research  Institute  for  the  Behavioral  and  Social  Sciences  has  under¬ 
taken  a  program  to  determine  the  attributes  which  make  eome  attack  pilots 
outstanding  performers  in  combat.  The  program  which  was  proposed 
consists  of  Che  following  three  inter-related  tasks;  (11  development  of 
an  attack  pilot  profile  from  survey  data  of  proven  performers;  (2l 
development  of  a  racing  form  to  evaluate  potential  attack  pilots;  and 
f3)  evaluation  and  assessment  of  AH-1  trainees  *  using  the  findings  of 
tasks  1  and  2. 


Currently  no  systematic  selection  of  candidates  for  AH-1  transition 
exists.  Many  trainees  do  not  want  to  be  attack  pilots  but  are  assigned 
to  fill  requirements  because  they  are  due  for  reassignment.  If  unit 
commanders  and  personnel  had  more  information,  a  better  fit  of  aviators 
to  transition  training  might  be  possible.  Tills  research  is  part  of  task 
2  and  was  done  to  determine  the  reliability  of  unit-level  ratings  of 
poccntial  AH-1  candidates. 


\ 
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OBJECTIVES 


The  primary  objectives  of  this  study  were  to  evaluate  the  incer-ratcr 
reliability  of  a  rating  form  dc.signcd  to  select  AH-1  transition  candi¬ 
dates  and  to  study  rater  acceptance  of  rating  forms. 

It  was  hypotheslaed  chat  CCSRA  pilots  In  cavalry  and  assault  units 
would  demonstrate  high  InCer-ratcr  rcliablltty  when  rating  non-AH-1 
qualified  aviators  in  their  units  «i  on  potential  for  AH-1  transition 
and  gunshlp  pilot  duties.  A  variable  of  interest  was  the  effect  of 
length  of  rater-ratee  acquaintance  on  inter-rater  reliability  iFreeberg, 
1960).  ^ 

Structured  interviews  of  tsxire  chan  50  attack  pilots  revealed  that 
about  70j  felt  that  they  could  predict  other  pilots  who  were  potential 
"ACEs"  (l.e.,  attack  pilots  who  perform  exceptionally  well  in  combat) 
without  observing  them  under  combat  conditions.  This  preliminary  research 
suggested  that  a  majority  of  attack  pilots  would  favor  use  of  rating 
forms  to  evaluate  candidates  for  AH-1  transition  in  contrast  to  a  general 
lack  of  acceptance  of  ratings  to  determine  leadership  potential  in  Army 
schools  (Medland,  Yates  and  Downey,  1974;  Downey,  1^5). 


AH,  attack  helicopter  «  CCBRA;  UH,  utility  helicopter. 


. . 


METHOD 


SAMPLE 

Ratets.  The  rateee  were  UJI-1  and  UH-l/OH-58  qualified  aviators  in 
six  troops  of  the  6th  ACCB,  one  troop  of  the  2d  Anaor  Division,  and  one 
troop  of  the  Ist  Cavalrs*.  For  some  units  all  eligible  ratees  were  used. 
For  those  troops  which  had  at  re  than  13  eligible  ratees,  a  random  pro¬ 
cedure  was  used  to  eliminate  individuals  and  reduce  the  number  of  ratees 
to  12.  The  modal  number  of  ratees  was  12  (see  Table  1).  Because  of 
anonymity  requirements  imposed  by  unit  commanders,  no  analysis  of  the 
lackgrounds  of  ratees  was  possible. 

Raters .  The  raters  consisted  of  all  the  available  AH-1  qualified 
aviators  in  the  eight  aviation  units.  There  was  considerable  variation 
in  the  number  of  raters  available  within  each  troop  (see  Table  1). 
Because  of  field  duty  assignments  many  AH-1  aviators  were  not  available 
for  participation  in  this  study.  However,  no  systematic  basis  for  non¬ 
availability  which  would  affect  this  study  was  apparent. 


PROCEDURE 

Several  troop  commanders  were  unwilling  to  have  their  personnel 
participate  in  this  study  unless  careful  procedures  were  used  to  Insure 
the  anonymity  of  raters  and  ratees.  This  requirement  was  satisfied  as 
follows :  The  raters  in  each  troop  were  scheduled  tt  nrrlve  at  the 
classroom  as  a  group.  Each  rater  within  a  group  was  given  a  packet  with 
a  randomly  preassigned  two-digit  number  as  he  entered  the  classroom. 

When  the  entire  troop  arrived,  the  list  of  names  of  ratees  for  the  troop 
was  placed  on  the  blackboard  and  a  set  of  three-digit  nundjers  v.*ere 
randomly  assigned  to  each  ratce.  The  raters  were  told  that  the  set  of 
numbers  on  the  board  corresponded  to  three-digit  numbers  appearing  on 
the  evaluation  forms  in  their  packets.  It  was  clearly  stated  that  no 
previous  connection  between  names  and  numbers  had  existed.  Racers  were 
then  instructed  to  consider  the  entire  list  of  individuals  before 
following  the  written  instructions  on  the  racing  forma.  lomwdlately 
after  each  troop  completed  their  ratings  the  names  and  ntoabera  on  the 
blackboard  were  erased.  No  records  of  ratees’  numbers  were  kept. 


RATING  SCALE 

The  rating  scale  ves  modeled  after  a  general  format  used  at  several 
Army  schools’  and  is  desired  to  encourage  dlecriminetion  among  ratees 


a 


For  example.  The  RANGER  Evaluation  Report;  FB  (RD)  Form  2, 


on  a  sec  of  desirable  characteristics  for  gunship/attack  pilots.  Hie 
attack  pilot  characteristics  were  identified  during  structured  interviews 
of  attack  pilots  with  combat  experience.  On  the  evaluation  form  the 
rater  is  instructed  to  consider  the  attack  pilot  characteristics  and 
assign  the  ratee  a  numerical  rank  (l.e.,  1-25  in  a  typical  group  of  25 
pilots)  representing  potential  for  success  as  an  attack  pilot.  The 
rater  Is  also  provided  space  for  a  2-3  sentence  word  picture  to  justify 
the  numerical  rating  assigned.  Additional  information  is  recorded  on 
where  the  rating  was  conducted  and  the  type  and  duration  of  the  relation¬ 
ship  between  rater  and  rate©.  Detailed  instructions  also  spell  out 
the  restrictions  Co  be  followed  when  rating  a  group  of  AK-i  candidates 
(see  Appendix  .\)  . 


FATING  FOKM  EVALUATION 

The  racing  form  evaluation  was  designed  to  ask  AH-1  qualified 
raters  the  following:  (1)  characteristics  to  add  to  or  delete  from 
c.he  list  on  the  racing  fora,  (2)  if  ratings  should  be  used  and  who  should 
do  the  racing,  and  (3)  open-ended  questions  about  why  they  do  or  don’t 
favor  ratings  and  potential  operational  problems  of  using  ratings. 


RESULTS  ANT)  DISCUSSION 

Coefficients  of  concordance  and  average  rho's  were  computed  for 
each  unit  to  determine  the  degree  of  agreement  among  raters.  Tlie 
results  in  Table  1  indicate  chat  inter-rater  agreement  was  very  high 
(p  ,01)  for  6  of  the  3  units  studied.  Although  statistical  signif¬ 
icance  was  not  quite  achieved  in  units  6  and  7  (p  -  .02),  the  number 
of  raters  involved  was  small.  The  overall  results  indicate  that  raters 
were  responding  to  essentially  the  satsc  characteristics  when  rating 
the  candidates  In  their  units.  This  does  not  mean  that  the  oh3er\'ed 
orderings  were  necessarily  correct,  or  based  on  the  ’'true*'  criteria 
for  a  successful  gunship  pilot;  however,  the  location  of  the  attack 
pilot  characteristics  on  the  rating  fora  itself  was  designed  to  make 
the  racing  criteria  explicit  and  reduce  the  influence  of  irrelevant 
variables. 

The  results  of  the  Rating  Fora  Evaluation  are  presented  In  Appendix 
3.  The  percentage  of  Che  sample  (if  over  5%)  who  reapondad  to  an 
alternative  or  who  oade  an  open-ended  cotauent  is  shown  on  a  copy  of  the 
rating  form  itself.  The  most  important  result  was  that  582  of  the 
respondents  felt  ratings  should  be  used  in  selecting  candidates  for 
COBRA  transition  training.  However,  many  raters  indicated  reservations 
about  the  validity  of  racings  because  of  the  influence  of  personal 
prejudices  and  friendships  (202)  and  the  adequacy  of  ratings  to  assess 
candidates  effectively  (15%) .  To  the  question  "\Aio  do  you  think  should 
rate  AH-1  candidates,”  only  20%  of  the  sample  responded  ’’eosmanding  and 
supervisory  personnel,”  while  53%  indicated  either  AH-1  qualified  pilots 
(29%)  or  other  pilots  in  the  unit  (29%) . 
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Another  variable  which  is  isaportant  in  peer  rating  research  is 
how  well  the  rater  knows  the  Individual  being  rated.  The  measure  of 
acsoclation  recorded  in  this  study  was  the  length  of  time  the  rater  knew 
Che  rated  officer.  The  relationship  between  median  duration  of  rater- 
ratee  acquaintance  vflthin  each  unit  and  the  statistical  reliability  of 
Inter-rater  agreement  is  shown  in  Table  2.  The  data  in  Table  2  suggest 
that  inter-rater  reliability.  In  these  units,  may  be  positively  related 
to  the  duration  of  association  between  raters  and  ratees. 


Table  2 

relationship  between  inter-rater  reliability  and 

DEGREE  OF  5li\TF.R-RATEE  ACQUAINTANCE  WITHIN  UNITS 


Unit  No. 


Median  Number  of 
Months  of  Rater- 
Ratee  Acqua  isitance 


Significance  Level 
of  Intcr-katcr 
Agreement 


P  <  .001 

p  <  .001 
p  <  .001 
p  <  ,001 
p  <  .01 
p  <  .02 
p  <  .02 


CONCLUSIONS 


AH-1  qualified  aviators  demcnstrate  a  high  degree  of  inter-rater 
reliability  when  rating  AH-1  candidates  in  their  units.  Statistical 
reliability  was  higher  w'ithln  those  units  \4iere  raters  and  ratees  had 
been  acquainted  longer. 

The  majority  (582)  of  All-1  qualified  raters  sampled  feel  that 
ratings  by  fellow  aviators  should  be  twed  in  selecting  candidates  for 
COBHA  transition. 


Many  raters  (20T)  had  reservations  about  the  use  of  ratings 
because  of  the  Influence  of  friendship  and  other  personal  biases. 
Another  group  felt  ratings  could  not  (8%)  or  oi|^t  not  (72)  be 
effective  in  Identifying  potentially  successful  AH-1  candidates 
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APPENDIXES 


APPENDIX  A 


ATTACK  PILOT  CANDIDATE  EVALUATiaN  FORM 


Instructions: 

1.  Evaluate  this  man  In  your  unit/class  in  terms  of  your  estimate  of  his 
potential  ability  to  become  a  successful  gunship/attack  pilot.  Determine 
where  you  think  he  would  rank  in  a  typical  group  of  25  pilots  (number  1 
the  highest  ranking,  25  the  lowest  ranking).  Consider  the  ATTACK  PILOT 
CHARACTERISTICS  below  prior  to  rating  each  man.  Consider  the  entire 
group  you  are  asked  to  evaluate  and  the  followingrestrictions  before 
beginning,  (a)  No  more  than  two  individuals  may  be  placed  in  1-5  column. 

(b)  no  two  individuals  will  be  assigned  the  same  rating  number.  Do  not 
rate  yourself. 

2.  Under  REMARKS,  write  a  2-3  sentence  word  picture  to  justify  the  numerical 
rating  you  assigned.  State  briefly  the  characteristics  (desirable  or  un¬ 
desirable)  of  this  man  that  impressed  you  most. 

3.  Your  ratings  will  remain  anonymous.  The  packet  you  picked  up  has  an 
ID  number  only  to  insure  that  you  followed  the  restrictions  when  rating. 
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DAY  WNTK  YEAR 


TACTICAL  KNOWLEDGE  AGGRESSIVENESS  CONFIDENCE 

PHYSICAL  STAMINA  SELF-DISCIPLINE  TEAIAt'ORK 


TIMELINESS  OF  ACTION  DRIVE  INITIATIVE 


COMBAT  INSTINCT  EFFECTIVE  MAP  USE  DEPENDABILITY 

EVALUATED  PERSON'S 
PRESENT  LOCATION. 
(Circle  one) 

7W 

UNIT 

TRA.NSITION 

TRAINING 

STAfiDING  WITHIN  A 

25 -MAN  GROUP 
(Circle  one) 

REUTIONSHIP  TO 
EVALUATED  PERSON 
(Circle  one) 

CO 

IP 

OTHER 

1  6  11  16  21 

2  7  12  17  22 

3  8  13  18  23 

4  9  14  19  24 
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APPENDIX  B 


RATING  FORM  EVALUATION 

1.  What  characteristics  would  you  add  to  the  list  on  the  rating  form? 

%  Responding 


Desire  17X 

Consson  Sense  17X 

Hanlal  Staialna  7X 

Maturity  5S 


2.  What  characteristics  would  you  delete  from  the  list  on  the  rating  fortnT 

%  Responding 

Coabat  Instinct  14* 

Physical  Staaina  82 

3.  Who  do  you  think  should  rate  AH-1  candidates? 


AH-1  Quali¬ 

All  other 

Commanding 
and  super¬ 

Instructor 

Senior 

fied  pilots 

pilots  in 

visory  per¬ 

pilots 

aviators 

only 

their  unit 

sonnel  only 

only 

only 

(292) 

(292) 

(202) 

(14%) 

(82) 

4.  What  problems,  if  any,  can  you  foresee  in  using  a  rating  form  like 
the  Attack  Pilot  Candidate  Evaluation  form  at  the  unit  level?  Use  the 
reverse  side  if  more  space  is  needed. 


%  RespondinR 

Personal  prejudices  and  friendships  will  invalidate  It  202 

Forts  can’t  Judge  qualifications  152 

5.  Do  you  feel  that  ratings  by  fellow  aviators  should  be  used  In  selecting 
candidates  for  COBRA  transition? 

YES  NO 

*  Responding  (582)  (42%) 

Briefly  irsdicate  In  the  space  below  why  you  feel  ratings  should/should 
not  be  used.  Use  the  reverse  side  if  more  space  is  needeo. 

Should  2  Responding  Should  Not  %  Responding 

Fellow  aviators  eorj  Ratings  are  biased  82 

aware  of  abilities  122  Ratings  are  no  good  82 

Ratings  nay  not  be  any  good  72 


